VIRTUAL RESEARCH CENTRE (VRC) 



Commentary on the OECD’s 
Programme for International 
Student Assessment (PISA) 

2009 Study 

by Bernie Froese-Germain 
CTF Researcher 
Research & Information Dept. 

January 201 1 



© Canadian Teachers’ Federation 

Federation canadienne des enseignantes et des enseignants 





Commentary on the OECD’s Programme for 
International Student Assessment (PISA) 2009 Study 


Background and Overview of PISA 

The Organisation for Economic Co-operation and Development’s Programme for 
International Student Assessment (PISA) is an international standardized test administered 
every three years to random samples of 15-year-old students in all 30 OECD member 
countries as well as a growing number of OECD partner countries. It is not designed on 
the basis of national curricula and programs but rather, PISA measures, assesses, and 
compares the extent to which 15-year-old students nearing the end of their compulsory 
education possess and demonstrate key knowledge and skills for their full participation in 
society. 

The OECD is an international organization dedicated to the market economy and, within 
education, to the development of a labour force that meets the needs of the market 
economy. It is important to note that PISA is less about assessing student learning, and 
more about how education systems are meeting the OECD goals. It is not a true measure 
of the quality of education for any student. 

To date there have been four cycles of PISA: 

• PISA 2000 - 43 countries participated in the assessment; subject focus was 
reading (primary assessment domain) 

• PISA 2003 - 41 countries participated; focus on mathematics 

• PISA 2006 - 57 countries involving approx. 400,000 students; focus on 
science 

• PISA 2009 - 65 countries participated (plus an additional 9 countries which 
joined PISA 2009 later and which will be reported on separately in a year’s 
time) involving over 500,000 students; focus on reading (minor domains - 
math and science) 

Key aspects of PISA 2009 are summarized below (excerpted from Education 
International’s preliminary analysis of PISA 2009): 

• The 2009 survey marks a new cycle of assessment of the three ‘domains’ 
that are assessed by PISA every three years - reading literacy, mathematics 
and science. While the main focus of PISA 2009 is on reading literacy (as in 
PISA 2000), the report also includes data on mathematics (focus of PISA 
2003) and science (focus of PISA 2006), providing for the first time the 
possibility for conducting a trends analysis and comparisons with the results 
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from previous cycles in reading since PISA 2000. Comparisons and trends 
analysis with previous cycles in mathematics and science are also made in 
PISA 2009, albeit to a lesser degree. As in the previous two cycles, a 
problem solving component is included to assess cross-curricular 
competences, in part being conducted in a digital version. 

Originally, PISA covered only OECD countries; however, it has gradually 
extended its scope to include OECD partner countries and other countries as 
well. Countries / provinces included in the PISA 2009 cycle that were not 
included in the previous cycle include: Albania, Dubai (United Arab 
Emirates), Kazakhstan, Macao, Panama, Peru, Shanghai Province (China), 
Singapore, Trinidad & Tobago. 

PISA 2009 covers three main assessment areas (so-called ‘domains’): 
science, reading and mathematical literacy. In each of these domains, tasks 
require students to demonstrate literacy (i.e. understanding of concepts), 
knowledge of the domain, competencies, and understanding of contexts and 
situations. On the basis of this methodology, PISA develops detailed student 
performance indicators and correlates them with background data about the 
students and schools, and then from those correlations the report draws 
policy conclusions. 

The results of PISA 2009 are published in five volumes covering: 

> student performance in reading (Volume I) 

> equity in learning opportunities and outcomes (Volume II) 

> student engagement in learning (Volume III) 

> resources, policies and practices at the school level, and what 
makes a school successful (Volume IV) 

> learning trends (changes in student performance since PISA 
2000) (Volume V) 

A sixth volume will be published in June 201 1 that focuses on digital reading 
capabilities. 

In PISA, students are selected to perform a 2-hour paper-and-pencil test. 
Over half a million students were randomly selected to participate in PISA 
2009, representing about 28 million 15-year-old students in the schools of the 
65 participating countries. At a country level, the coverage of this year’s 
report is presented as a representative sample ranging between 3,500 and 
50,000 students in each country. Most federal countries also draw regional 
samples. These tasks require students both to develop their own answers 
and to fill in multiple choice questions. 
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• In addition, PISA asks students to provide information on their personal 
background, learning habits and attitudes, motivation towards and 
engagement in learning, as well as about school climate and family 
background. Performance differences are also assessed through information 
gathered from parents, principals and system leaders on school policies, 
practices, resources and institutional factors. Specifically, school principals 
complete a special questionnaire about their schools’ characteristics (size, 
demographic composition of the student population, etc.), resources, 
institutional factors and learning environment, and parents complete a 
questionnaire about their home and family background. As noted in PISA 
2009 information has also been collected from “system leaders” i.e. 
policymakers. 

• A significant part of the analysis is devoted to the correlation of background 
data, including gender, with student performance. However, PISA does not 
contain a special teachers’ questionnaire, revealing that the views of 
teachers are not taken into account, while they play a key role in the learning 
process. 

According to the OECD, key features of PISA include: 

• its policy orientation, with design and reporting methods determined by the 
need of governments to draw policy lessons. 

• its contextualization within the system of OECD education indicators, which 
examine the quality of learning outcomes, the policy levers and contextual 
factors that shape these outcomes, and the broader private and social 
returns to investments in education. 

• its breadth of geographical coverage and collaborative nature, with more 
than 60 countries - covering roughly nine-tenths of the world economy - 
having participated in PISA assessments to date, including all 30 OECD 
member countries. 

• its regularity, which will enable countries to monitor their progress in meeting 
key learning objectives. 

These combined features of PISA - policy orientation, contextualization within the OECD 
education indicators, and the breadth and regularity of testing - as well as its focus on 
providing international comparative data in an era of data-driven policymaking, make PISA 
a powerful instrument for shaping education policy among OECD countries and beyond. 

Canada has participated in PISA since its inception in 2000. Our involvement in PISA is 
administered through a partnership of the Council of Ministers of Education Canada 
(CMEC), Statistics Canada, and Human Resources and Skills Development 
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Canada (HRSDC). Approximately 23,000 15-year-old students from about 1 ,000 schools 
across the ten provinces took the PISA 2009 test in April and May 2009. The assessment 
was administered in both English and French according to the respective school system. 
Oversampling by provinces of the students to be tested allows comparisons to be made 
not just with other provinces and with Canada as a whole, but also with other countries. 

Part of the rationale for harmonizing PISA with Canada’s national testing program, the 
Pan-Canadian Assessment Program (PCAP), is to allow the provinces to examine and 
compare patterns of performance between their PCAP and PISA assessments, and 
validate these results with the results of their own provincial assessments. 


Summary of PISA 2009 Results for Canada 

The following is a summary of the Canadian results of the OECD PISA study released on 

December 7, 2010: 

Reading (major domain) 

• Canadian students continue to perform well in reading in a global 

context (among 65 countries only four countries outperformed Canada while 
three countries had similar performance to Canada). 

• There is significant variation in performance between Canadian provinces in 
reading. 

• Canada continues to demonstrate strong performance and high equity in 
reading performance. 

• Canada had a larger proportion of high achievers and a smaller proportion of 
low achievers compared to the OECD average. 

• In most Canadian provinces, students in minority-language school systems 
had lower reading performance than students in majority-language school 
systems. 

• Females continue to outperform males in reading. 

• Reading performance in five of the ten Canadian provinces decreased 
between 2000 and 2009. 

• Canada’s proportion of high achievers in reading decreased between 2000 
and 2009. 

• Between 2000 and 2009, the gender gap in reading remained stable in Canada and 
across nine provinces but was significantly reduced in New Brunswick. 
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Mathematics and science (minor domains) 

• Canadian students performed well in mathematics and science. 

• Most provinces performed at or above the OECD average in mathematics 
and science. 

• Canadian males outperformed females in mathematics and science. 

• Canada has more equity in performance compared to most OECD countries. 

• In most provinces students attending majority-language school systems 
outperformed students who attend minority language systems. 

• Canadian students’ performance in mathematics and science remained 
stable over time. 

A second report on the Canadian PISA results, to be released in Spring 201 1 , will examine 
the relationship between student background characteristics, school factors, and student 
engagement with reading achievement. 
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Commentary 

These are some of the major issues raised in Education International’s preliminary 
analysis of the overall PISA 2009 study (the following are excerpts from the analysis): 

• PISA has serious limitations. The study does not cover the full curriculum, 
focusing on a narrow set of subject areas, neglecting such important 
domains as the arts, humanities and social sciences. 

• Moreover, as noted above PISA is not designed on the basis of national 
education goals, curricula and programs, but it applies its own innovative 
methods to assess literacy and competencies in reading, mathematics and 
science. PISA provides only a snapshot of the selected group at one point in 
time during the life of the school. While PISA can provide a picture, causal 
conclusions should not be drawn. As the OECD itself will explain, PISA can 
only reveal what conditions are found in situations of strong or weak 
performance. Therefore, PISA cannot be considered as the ultimate 
measurement of the quality of education systems. Nevertheless, the 
impression that PISA leaves on the current education policy debates is 
enormous. 

• PISA is intended to provide policy guidance to governments. Predictably, the 
reports will argue that the results show how countries can learn from each 
other about how to set and achieve measurable goals achieved elsewhere. 
PISA claims that its tests are designed to support governments in training 
their students to “...deal with rapid change, to find jobs that have not yet been 
created, to use technologies that have not yet been invented, to solve 
problems which have not yet arisen” (Presentation made by Michael 
Davidson (OECD) on December 1, 2010, during the EI/TUAC - OECD 
consultation session in Paris). Such claims are debatable, as there is little 
proof that performance in PISA tests is a relevant predictor for individuals’ 
success in the future. 

• PISA presents the performance of different students in different 
countries (each cycle brings in new countries and territories) at different 
times, and in different political, social and economic circumstances. The 
collection of 2009 PISA data overlapped with the deep economic crisis and 
recession affecting many OECD and partner countries, yet the PISA analysis 
does not take this broader context into account. 

• Students’ achievements in PISA are presented in terms of the percentage of 
students reaching six different proficiency levels. These levels are 
constructed on the basis of students’ ability to use measured competencies. 

In PISA 2009, the OECD argues that the biggest improvements of “country 
performance” (i.e. rise in ranking) are achieved by narrowing the proportions 
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of students performing in the lowest levels, rather than increasing the top 
levels. El believes this conclusion argues for greater equity in education, and 
for boosting resources for students from disadvantaged backgrounds. There 
is a danger, however, when governments push competition between schools, 
of perverse effects such as the exclusion of lower performing students and 
those with learning difficulties. Overemphasis on narrow indicators of 
performance could lead to teachers coming under pressure to neglect other 
important aspects of education. The PISA test does not tell us what the 
lower performing students may be good at - only that they do not write PISA 
tests well. 

• Another important aspect of the PISA 2009 results is gender differences. As 
in previous cycles, girls are revealed to outperform boys in reading in 
general, in this case doing better in every country surveyed - on average 
girls were one grade level ahead of boys in reading. Boys performed better 
than girls in mathematics although less explicitly (there will be exceptions), 
while both sexes show similar results in science. Particular attention could 
be given to the differences in reading different types of texts (magazines, 
newspapers, fiction and non-fiction) and correlations between reading 
preferences and performance. 

• El stresses that while PISA data do show correlations between various 
performance and contextual variables, it does not however establish a causal 
relationship. As such all arguments and policy conclusions should be 
countered by the argument that we don’t know if features presented in PISA 
actually explain the variation in student performance, or are they just side- 
effects of other factors beyond the scope and scale of the study. 

• On a more positive note PISA 2009 again makes a strong claim about a 
positive correlation between equitable outcomes across the education 
system and average results. 

On the latter point, the best school systems are the most equitable, and Canada is notable 
in this regard. Christa Freiler, Director of Research and Strategic Initiatives at the 
Canadian Education Association, observes that “a major ‘good news’ story behind the 
latest PISA results is that Canada continues to be marked by high achievement and high 
equity in education.” She goes on to say that, 

This means that the impact of socio-economic status is relatively small, and 
the gap between the high achievers and low achievers is also small, 
compared to most other countries. This is the distinguishing feature of 
Canada’s education system and, arguably, more important to the social and 
economic future of young people and Canada as a whole than small changes 
in overall standing (i.e. whether we are 3rd, 4th or 5th). 
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Why is equity in education so important? First, because it means that, 
generally speaking, all children in Canada benefit from good schools and 
good educations, regardless of their family’s socio-economic status or 
immigration status. Why this is important for children is obvious since it 
affects their current educational experiences and their future prospects. 

Second, and perhaps less obvious, educational equity is important because it 
relates to the overall equality in a society. We now know that equality 
benefits everyone in a society, not just those at the bottom. In their book, 

The Spirit Level: Why Equality Is Better For Everyone, Richard Wilkinson and 
Kate Pickett demonstrate that health and social outcomes are considerably 
worse in more unequal countries. They found that this is true for physical 
health, mental health, drug abuse, education, imprisonment, obesity, social 
mobility, trust and community life, violence, teenage pregnancies, and child 
well-being. They conclude that, the smaller the social and economic 
inequality between people, the better it is for everyone. 

How do we explain Canada’s high level of educational equity? A common 
answer, particularly when contrasting Canada with the United States, is that 
it is a combination of factors, perhaps most importantly because we have 
better income programs, [a] social safety net, and [a universal publicly- 
funded] health care system. Last week UNICEF released a report that shed 
some new light on this explanation. The Children Left Behind looked at 
inequality in child well-being in three areas: material well-being (includes 
family income and housing); educational achievement; and physical health. 

The results were mixed. In material well-being, Canada ranked 17th out of 
24; in health, 9th out of 24; and in education, 3rd out of 24. The educational 
finding prompted the UNICEF spokesperson to observe “we are doing 
something right”. Significantly, it is our education system and our schools 
that are doing something right, since Canada’s record when it comes to 
income, health and housing inequality, as well as child and family poverty 
rates, is considerably less stellar. 

In a presentation at the CTF National Staff Meeting in November 2010, El Research 
Coordinator Guntars Catlaks situated education reform in an emerging global policy 
landscape characterized by fiscal and budgetary consolidation; pressure to reduce 
government debt and deficit; austerity measures in public services including education; 
growing reliance on public-private partnerships (P3s) for funding; ‘flexibilization’ of working 
conditions (individual versus collective contracts); and efficiency and effectiveness (“doing 
more with less”). This is the daunting landscape facing teacher organizations going 
forward. 
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PISA has become the OECD’s flagship study. It continues to expand as more 
countries (and states and provinces/entities within countries) participate with each new 
cycle - as Catlaks notes, the prevailing attitude is that, “if you’re not in PISA, you don’t 
count”. 

And while there are other international studies and surveys on education such as PIRLS 
(Progress in International Reading Literacy Study), TIMSS (Trends in International 
Mathematics and Science Study), and IALS (International Adult Literacy Survey), these are 
overshadowed by PISA, essentially ignored by the media’s obsession with PISA rankings. 

Education International’s analysis of the media coverage of PISA 2006 found little in the 
way of explanation or analysis of the results, with an emphasis on reporting the rankings 
and advocating for education reforms with reference to PISA. The focus on rankings by 
the media and politicians has resulted in the quality of education being reduced to a 
simplistic matter of performing better or worse. El notes that the PISA study has come to 
be perceived by policymakers, media and general society as a proxy for a broader deeper 
analysis of education systems, in part due to the rankings of countries according to their 
average results presented in the form of league tables. PISA is in fact too narrow to be a 
proxy. 

Catlaks notes that the “promise of PISA” is that it “overemphasizes the fact that national 
income and expenditure does not directly affect the attainment of good results .... This is 
one of the crucial presumptions to keep policymakers and the broader public interested, as 
they are promised to find out how to achieve better results without higher investments". 

The promise of improving achievement without additional educational spending is welcome 
news to governments in a period of fiscal restraint. 

Future cycles of PISA will see a gradual switch to computer/web-based assessment 
instruments, and the addition in 201 2 of a new assessment domain - financial literacy. 

The latter clearly demonstrates the narrow goals driving PISA. 

In addition, PBIT (or PISA-based international testing), described by Catlaks as “PISA a la 
carte”, is a pool of 7,000 PISA-style items developed by the OECD, available for countries 
to use in their own test construction. 

The OECD among other international organizations has been particularly influential in 
promoting the concept of education systems as providers of ‘human capital’ for the 
globalized economy, through mechanisms such as PISA and the annual Education at a 
Glance reports (providing comparative statistics on education systems). 

Sotiria Grek (2009) suggests that “the OECD’s greatest impact has been in relation to its 
Indicators agenda, including PISA, and its role in constructing a global educational policy 
field through governance by comparison" [emphasis in original]. 
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In effect, the act of comparing is as much a method as a form of policy in itself - this is 
education governance reduced to comparing numbers, and subsequently borrowing or 
copying policy directions from countries with higher numbers. 

Research by Education International on PISA 2006, documented in a recent CTF report 
called The OECD, PISA and the Impacts on Educational Policy, has found that PISA 
distorts educational policies as it has “encouraged governments to adopt an overly 
‘economic’ approach to education”. This research demonstrates how national 
governments have instituted educational reforms based on increasing efficiency and 
competitiveness, with reference to the latest PISA results as a source of legitimacy for their 
actions. 

In this regard the conclusion of the report on the Canadian results of the PISA 2009 
study (p. 37) is revealing, containing as it does many references to the importance of 
achieving and maintaining high skill levels among students as a determinant in Canada’s 
future economic success and prosperity in a global economy. 

The author of the El study, Laura Figazzolo, states that: 

In fact, what makes PISA different, and more dangerous than the other 
international comparative surveys in education, is its clear policy orientation 
led by the principle of increasing school efficiency. This renders it a powerful 
tool for political influence, as the OECD is able to exert a sort of peer 
pressure and “soft governance” on national governments, by virtue of its 
status as an authoritative impartial source of evidence. 

Indeed, in little more than a decade, the impact that the first three PISA cycles has had on 
shaping policymaking in education to meet economic objectives leads one to conclude that 
PISA is arguably a form of ‘high-stakes’ testing. In this case the stakes are high with 
regard to PISA’s observed impact on high-level policy and program decisions made by 
national governments and governments at other levels, through the auspices of a highly 
influential organization. 

On the challenges to the teaching profession posed by large-scale external assessment, 
ETFO’s Vivian McCaffrey observes that PISA and the OECD country reports that analyze 
the results (Canada’s country report on PISA is jointly produced by Statistics 
Canada/CMEC/HRSDC) have led to an expansion of national assessments in Europe and 
elsewhere, and intensified the testing culture around the world including in Canada. 

In addition to PISA, the OECD is developing a number of new international surveys on 
education. Of particular interest to teacher organizations is the Teaching and Learning 
International Survey (TALIS). 
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The OECD website describes TALIS as, 

the first international survey to focus on the learning environment and the 
working conditions of teachers in schools and it aims to fill important 
information gaps in the international comparisons of education systems. 

TALIS offers an opportunity for teachers and school principals to give their 
input into education analysis and policy development in some key policy 
areas. 

TALIS looks at key issues affecting the work of secondary school teachers: 

• professional development 

• teacher beliefs, attitudes and practices 

• school leadership 

• teacher appraisal and feedback 

Results from the first TALIS survey involving 23 OECD and partner countries in four 
continents were published in June 2009. Canada was not among the participating 
countries for this first cycle. The OECD is in the process of developing a proposal for a 
second round of TALIS. It has invited all OECD member and partner countries including 
Canada to participate in a priority-rating exercise intended to guide the content of the next 
round of TALIS. 

Education International argues that the TALIS findings, with regard to teacher appraisal 
and feedback in particular, could be used to strengthen the argument for crafting systems 
of performance-based pay for teachers. 

Another concern is the potential linkage between TALIS findings and PISA results in terms 
of teacher evaluation being based on student performance on PISA. Guntars Catlaks 
warns that, 

There could be a potentially “dangerous” connection between TALIS and 
PISA. The emphasis on individual teachers and their style of teaching, 
beliefs, cooperative attitudes, and, above all, “effectiveness” can indeed be 
linked to how these teachers’ students perform in PISA, or PISA-like 
assessments, with dangerous consequences for individual teachers whose 
students do not perform high enough. In other words, could individual 
teachers be evaluated according to how well their students perform in PISA- 
type assessments? 

Indeed, subsequent rounds of both PISA and TALIS will be conducted in the same time 
frame -201 2/201 3. 
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Given PISA’s growing international profile and influence, and in an attempt to address El’s 
concern “that PISA data should be used more constructively”, the Education International 
Research Institute commissioned an expert paper on alternative models for analysing and 
representing countries’ performance in PISA. Describing PISA as “an OECD 
triumph” (which it undoubtedly is from an OECD perspective) as it “provides governments, 
academics and voters with rich information about their education systems”, the paper’s 
author, Peter Mortimore (2009), proposes that PISA could be “further enhanced” by doing 
the following: 

• shifting the aims of PISA from a snapshot of national achievements to a more 
nuanced interpretation of countries’ strengths and weaknesses in their 
development of lifelong learning 

• refocusing on how schools and school systems could promote achievement 
and increase the equity of their educational outcomes 

• widening the basis of assessment though inclusion of knowledge from 
geography and social science domains 

• involving teachers in the design and development phases of the PISA cycles 

• extending the methodology to include a longitudinal element 

• reconfiguring to an alphabetical order the minority of tables in the PISA report 
currently presented in rank positions 

While these recommendations for change may well be necessary, they may not be 
sufficient to counter the powerful trend of using PISA results mainly to rank order 
countries. 

According to Education International, while PISA reveals interesting data on correlations 
between the performance of students in reading, science and mathematics, their 
socioeconomic backgrounds, and the organization of schools, it is based on a relatively 
simple set of questions posed in a 2-hour paper-and-pencil test, answered by a sample of 
students drawn from one particular age-group (15-year-olds). As such it conveys neither 
the complexity nor the breadth of education systems, nor does it portray anything close to 
the total picture of education quality in any country. 
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